it would be nice if we could run a given intermittent test (say every new one) through test-verify where it would output the failure rate (intermittent rate). For bonus points, be able to "backfill" this job, so it would run the same test(s) on previous commits and you could easily see the failure rate over a time window. This requires us to have an easy way to input a testname into the parameters for the test-verify job. Then we could edit/retrigger or add-new test-verify, then edit/retrigger the job so it would gather the data. it is unknown if this would actually reproduce failures. Having 100+ data points would help determine if we could reproduce failures like this.

+1 for backfilling and otherwise making it easier to run test-verify against a specified test on a specified push. I am unsure about trying to produce a failure rate. TV normally stops once a failure is detected, consistent with the original goal of finding and reporting intermittent test failures as soon as possible. Also, for some suites, TV runs in 4 steps/modes: --repeat 20, 5x --repeat 1, chaos --repeat 20, chaos --repeat 1, etc. You might need to generate and compare a failure rate for each step.

I am thinking of running in a custom mode --repeat 100. This mode would have to allow for multiple failures; Maybe better would be to: 5x --repeat 20 that would allow for up to 5 failures and up to a max of 100 tests without modifying too much.

here is an experiment with very little code changes: https://treeherder.mozilla.org/#/jobs?repo=try&revision=580cd7f8a1452e74e90ab8063e5b6340c89f97c1&filter-searchStr=test-verify%20linux What I did was: * create regular test-verify job with random test changed * find a stockwell bug that fails on the platform, in this case bug 1459789 * edit+retrigger job from treeherder/taskcluster tools to add |MOZHARNESS_TEST_PATHS: 'path/test.html'| (I also changed symbol to TVi * profit this MOZHARNESS_TEST_PATHS works as a method for running a single test or a set of tests from |./mach try| It worked out such that this was a simple integration into test-verify. For the purposes of this bug, how can we move forward: * assuming we like this approach, get it landed * build a tool to make it easier to give a full test path and run it on test-verify ** watch out for web-platform-tests or reftests which might not have the full test name * edit test verify to have a different mode when running with MOZHARNESS_TEST_PATHS, maybe something where it just runs the test 20 times by itself and repeat 5 times (although the above scenario shows it fails only in chaos mode). :gbrown, what do you think?

I like it!

a simple change here. I also cleaned up a bit in desktop_unittest.py to support test-coverage better.

as a note, this is part 1- there is still a lot of work to do for making this a better process and making it easier to find a failure rate.

Comment on attachment 8976172 [details] [diff] [review] support MOZHARNESS_TEST_PATHS for test-verify Review of attachment 8976172 [details] [diff] [review]: ----------------------------------------------------------------- This looks fine. Will MOZHARNESS_TEST_PATHS give us better try support for free? ./mach try -u test-verify-e10s <path> ?? I would like to see android and wpt kept in-sync (have similar capabilities and behavior at any point in time). Looking ahead, I still do not like the idea of calculating failure rate. Nor do I want to see verify run a different number of times or skip steps in certain modes: I think it should be consistent whether run locally, on try, automatically because a test was modified, or because of MOZHARNESS_TEST_PATHS. Keep it simple: verify PASS means "this probably won't produce an intermittent", FAIL means "this test will probably fail intermittently (or cannot be run stand-alone, or cannot be repeated)".

Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/dcd341823c77 test-verify should support MOZHARNESS_TEST_PATHS. r=gbrown

now that we can run an arbitrary test, all we need is a tool that is part of treeherder/taskcluster that adds a test-verify job with the appropriate parameters. I don't think there is anything specific to do here.